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1. Field of the Invention 

[001] The present invention is related to image processing and, in particular, to 
an image processing system having an RGB to YUV conversion. 

2. Summary of Related Art 

[002] There is current interest in all aspects of acquiring and transmitting 
multimedia images over various media. The speed of acquiring and encoding data is 
crucial to the successful real-time transmission of multi-media images over various 
media. Further, in some applications, the speed of encoding data is also crucial for 
acquiring and encoding data for multimedia images. Video data can be digitally acquired 
with a multi-pixel array of video sensors. These sensors are usually arranged in an RGB 
format, and therefore data is obtained from the video sensors in RGB format. 

[003] A recently developed encoding scheme for multi-media images is 
MPEG-4. Typical implementations of MPEG-4 utilize YUV video data. Therefore, a 
data conversion from RGB to YUV format is performed. 

[004] Therefore, there is great interest in conversion of RGB to YUV format. 
Further, there is a great interest in high-resolution conversion of RGB data into YUV data 
for encoding and transmission. 

SUMMARY OF THE INVENTION 

[005] In accordance with the present invention, processing of data from RGB 
format into YUV data appropriate for encoding for transmission is presented. In 
accordance v^ith the invention, RGB data is translated into Y data and UV data in separate 
filter paths. In some embodiments, interpolation of RGB data into separate color planes 
in the lumina filter path includes filters with a lower number of taps than does the 
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interpolation of RGB data into separate color planes in the chroma filter path. Filters in 
each path can then be optimized for the characteristics of that path. 

[006] Consequently, a RGB to YUV video filter according to the present 
invention includes a lumina filter path coupled to receive blocks of RGB formatted data 
and generate a block of lumina data; and a chroma filter path coupled to receive the 
blocks of RGB formatted data and generate blocks of chroma data, wherein the lumina 
filter path and the chroma filter path operate in parallel. 

[007] In some embodiments, the lumina filter path includes an interpolation 
filter and an RGB to Y conversion. In some embodiments, the interpolation filter 
includes a 5-tap vertical filter and a 5-tap horizontal filter. Additionally, the lumina filter 
path can include a gamma correction, an edge enhancement filter, and a spatial filter. 

[008] In some embodiments of the invention, the chroma filter path includes an 
interpolation filter and an RGB to UV conversion. In some embodiments, the 
interpolation filter includes a 7-tap vertical filter and a 7-tap horizontal filter. Further, the 
chroma filter path can include a color correction block, a gamma correction block, and a 
core chroma correction block. 

[009] Therefore, a method of converting RGB formatted video data to YXJV 
formatted video data according to the present invention includes receiving a block of 
RGB video data; filtering the block of RGB video data through a lumina filter path; and 
filtering the block of RGB video data through a chroma filter path, wherein the lumina 
filter path and the chroma filter path are parallel paths. 

[010] In some embodiments, a 24x24 block of pixel data is received into the 
lumina and chroma filter paths. In some embodiments of the invention, a black-clamping 
adjustment is performed on the block of pixel data. In some embodiments, the block of 
pixel data is white-balance adjusted. 

[Oil] In some embodiments, filtering the block of RGB video data through a 
lumina filter path includes interpolating the RGB data to form red, green and blue color 
planes with interpolated values associated with the appropriate color in each pixel 
location of the red, green and blue color planes, and generating the lumina Y data block 
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from the red, green and blue color planes. Gamma correction, edge enhancement, and 
spatial filtering may also be performed on the lumina data block. 

[012] In some embodiments, filtering the block of RGB video data through a 
chroma filter path includes interpolating the RGB data to form red, green and blue color 
planes with interpolated values associated with the appropriate color in each pixel 
location of the red, green and blue color planes. In some embodiments, vertical filter and 
a horizontal filter are applied to the RGB video data block. In some embodiments, the 
vertical filter and the horizontal filter each include a 7-tap filter. U and V data blocks can 
then be determined from the red, green and blue color planes. Other filters, such as color 
correction, gamma correction, and core chroma correction can also be applied to the data. 

[013] It is to be understood that both the foregoing general description and the 
following detailed description are exemplary and explanatory only and are not restrictive 
of the invention, as claimed. These and other aspects of the invention are further 
discussed below with respect to the following figures. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[014] Figure 1 shows a block diagram for acquiring and processing data from a 
video array into MPEG-4 format according to the present invention. 

[015] Figure 2A illustrates RGB pixel format for the sensor shown in Figure 1. 

[016] Figure 2B illustrates the sensor array shown in Figure 1 . 

[017] Figure 3 A through 3T illustrate an embodiment of a pixel correct process 
according to the present invention. 

[018] Figure 4 illustrates an embodiment of an Enhanced CCD/CMOS Image 
Process (ECIP) block according to the present invention. 

[019] Figures 5 A through 5E illustrate the interpolation of RGB data for the 
luminance filter path shown in Figure 4. 

[020] Figure 6A illustrates the filtering for edge enhancement in the lumina 
filter path of the ECIP block shown in Figure 4. 
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[021] Figure 6B illustrates the operation of the spatial filter of the lumina filter 
path of the ECIP block shown in Figure 4. 

[022] Figure 7 A shows a block diagram of the interpolation block of the 
chroma filter path of the ECIP block shown in Figure 4. 

[023] Figures 7B through 7E illustrate the vertical and horizontal interpolation 
for the chroma filter path of the ECIP block shown in Figure 4. 

[024] In the figures, elements having the same designation perform the same or 
similar functions. 

DESCRIPTION OF THE EMBODIMENTS 

[025] In conventional RGB to YUV conversion systems, an interpolation of the 
red, green and blue data in the original pixel data is made in order to project color values 
for pixels in the sensor array that are not sensitive to that color. From the red, green and 
blue interpolated data, lumina and chroma values are generated. However, these methods 
do not take into account the different filtering and resolution requirements for lumina 
versus chroma data. Consequently, systems do not optimize the filtering or interpolation 
process based on whether lumina data is being produced or chroma data is being 
produced. In accordance with the present invention, separate and parallel lumina filter 
paths and chroma filter paths allow conversion firom the original data in a more optimum 
fashion for both lumina and chroma data. 

[026] Figure 1 shows an image processing circuit 100 according to the present 
invention. Image processing circuit 100 includes a sensor array 101 for detecting an 
image and a memory 104 for storing image data while that image data is being processed. 
Sensor interface 102 reads data fi-om sensor array 101 and stores that data in memory 104 
for fiirther processing. Black clamping 103 determines the black-level offset for the 
pixels in sensor array 101. Defective pixel correct 105 processes data that has been 
written into memory 104 by sensor interface 102. The Enhanced CCD/CMOS Image 
Process (ECIP) module 1 06 provides image data processing and conversion to YUV 
format. Filter 107 provides further filtering of data before the data is encoded with 
MPEG-4 encoder 109. Once encoded, the data can then be transmitted. 
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[027] In accordance with the present invention, ECDP module 106 includes 
separate parallel filtering paths to produce lumina data, Y data, and chroma data, U and V 
data, from the RGB formatted data from sensor array 101. The separate parallel filtering 
paths include separate interpolations of the RGB data into red, green and blue color 
planes to take better advantage of the eye's greater sensitivity to spatial variation rather 
than color variation. 

[028] Some embodiments of the invention may implement the filters and other 
processes here in h2irdware or with a microprocessor. Although a microprocessor 
implementation provides greater versatility in modifying algorithms, the hardware 
implementation provides faster data processing speeds. Therefore, in the present 
disclosure, block diagrams are provided for a hardware implementation. A software 
implementation, however, can also be produced fi'om this disclosure. 

[029] Memory 104 can be any memory large enough to hold the video data 
fi-om sensor array while it is being processed and any intermediate results calculated by 
the various processing blocks. Memory 104 can include any type of RAM, including 
dynamic or static RAM. In some embodiments, DRAM is utilized. Memory 104 may 
also include fiash memory for non-volatile storage of various programmable parameters 
and values that may be utilized over a long period of time. For example, the 
identification of invalid or non- function pixels in sensor array 101 may be stored in a 
flash memory portion of memory 1 04 and may be downloaded during operation into other 
portions of memory 104 for faster access. In some embodiments, memory 104 includes 
128 Mb or 256 Mb DRAM. Some embodiments include larger memories. 

[030] Sensor array 101 is typically a RGB format sensor array, such as that 
shown in Figure 2 A. As shown in Figure 2 A, sensor array 101 is arranged as a series of 
pixels that are light sensitive to green, red or blue light. The format shown in Figure 2A 
is referred to as the "Bayer" format. In the Bayer format, each line and each column of 
pixels are alternatively sensitive to green and either red or blue. A 2 by 2 array of pixels, 
for example, would have two green sensitive pixels and one red sensitive pixel and one 
blue sensitive pixel. The active pixel array can include any number of pixels in the RGB 
format. Such sensor arrays are used in charge-coupled devices and CMOS sensors. 
Charge coupled device vendors include Sony, Panasonic, Kodak, Sharp, and Sanyo. 
CMOS sensors are produced, for example, by Agilent, and OmniVision. 
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[031] Data is read from the pixel array of sensor array 101 by sensor interface 
102. Typically, sensor interface 102 serially reads the time-integrated value for each of 
the pixels in sensor array 101 and writes that data into memory 104. In some 
embodiments, data read into memory 104 is digitized to 10 bits. In some embodiments, 
numerical values for 12 pixels are stored in a 128 bit (16 word) area of DRAM 104. 
Since typical DRAMs are word addressable, pixel data, which is often 10 bit unsigned 
data, is packed into 16 word segments for storage in a DRAM section of memory 104. 
Other function blocks that access memory 104 to retrieve or store data, then, need to pack 
and unpack data into memory 104 in the same way. In this way, for example, 24 by 24 
pixel blocks of data can be stored in memory 104 for later processing. 

[032] As shown in Figure 2B, sensor array 101 typically includes an active 
region 201 and a dark region 202. Data from dark region 202 is averaged in black 
clamping 103. For each channel in the 2x2 pixel array (i.e., the Go, B, R and Gi 
channels), an average of the values from pixels in the dark region 202 is calculated. That 
data can be stored as dark-value offsets in memory 1 04, or in some embodiments stored 
in registers in black clamping 103 for use in a later offset correction. 

[033] Defective pixel correct 105 fills in the values for inoperable pixels in 
sensor array 101. In some embodiments, the particular pixels that are considered to be 
defective are identified and that identification is stored in memory 104. In some 
embodiments, that identification may be supplied by the manufacturer of sensor array 
101 . In some embodiments, a determination of the defective pixels can be determined, 
for example, by selectively exposing sensory array 101 and measuring the response of 
each of the pixels in sensor array 101. For example, defective pixels can be determined 
by exposing sensor array 101 to white light for a particular period of time (the exposure 
time) and monitoring the output signal from each of the red, green, and blue pixels, which 
typically is an integrated value indicating the total amount of white light incident on the 
pixel for the particular period of time. 

[034] Figures 3 A through 3T illustrate correction for bad pixels. In general, the 
defective pixel is corrected based on the values of neighboring pixels of the same type. 
The expected value for an invalid pixel can be calculated by utilizing the values of the 
functioning pixels that are nearest or next-nearest neighbors of that pixel. For example, in 
Figure 3A the red pixel designated with the 0 symbol is an invalid pixel. The value for 
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that invalid pixel can be replaced utilizing the value of the surrounding red pixels that are 
circled. In one type of correction, the median value of the circled pixels is utilized as the 
value of the defective pixel. In some embodiments, the average value of the circled 
pixels can be utilized as the value of the defective pixel. In some embodiments, the type 
of correction that is used for each of the defective pixels identified can also be stored in 
memory 104 and may be user programmable. 

[035] In Figure 3A, a single red pixel at the center of a 5x5 block of pixels in 
the pixel array of sensor array 101 is defective. As was discussed above, the numerical 
value that replaces the value for the defective red pixel can be calculated by the four 
surrounding nearest-neighbor red pixels in sensory array 101. Figure 3B shows the 
similar case with a defective blue pixel. The numerical value for the defective blue pixel 
is replaced with a value calculated from the four nearest-neighbor blue pixels surrounding 
the defective pixel. As discussed above, the numerical value for the defective pixel can 
be either the average of the surrounding blue pixels or the median of the blue pixels, 
depending on the type of correction. 

[036] Figures 3C through 3 J illustrate situations where the 5x5 block of pixels 
includes a second invalid pixel of the same color as the identified defective pixel. Figure 
3C, for example, illustrates a defective red pixel where the nearest-neighbor red pixel two 
rows above the defective crystal is also defective. In that case, the numerical value of the 
defective pixel in Figure 3C can be corrected utilizing numerical values from the three 
non-defective nearest-neighbor pixels (shown circled) and the two next-nearest-neighbor 
pixels closest to the second defective pixel. The two next-nearest-neighbor pixels can be 
weighted by compared with the nearest-neighbor pixels in the calculation. Again, the 
weighted mean or average of the numerical values of the pixels shown circled in Figure 
3C can be utilized to replace the numerical value of the defective pixel at the center of the 
5x5 block of pixels shown in Figure 3C. Figure 3D shows the same correction for a 
similarly affected blue pixel. Figures 3E through 3 J illustrate corrections of the defective 
pixel (which may be red or blue) at the center of the 5X5 pixel array with various 
orientations of a second defective pixel of the same type as the pixel being corrected. 

[037] Figures 3K and 3L illustrate correction of defective green pixels. In 
Figure 3K, the center defective green pixel is from a different channel than the defective 
green pixel illustrated in Figure 3L. With no other defective green pixel as a nearest- 
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neighbor to the defective green pixel, the numerical value representing the defective 
green pixel can be replaced with either an average or the me£in of the numerical values of 
the surrounding nearest-neighbor green pixels. 

[038] Figures 3M through 3T illustrate various configurations of corrections 
where one of the nearest-neighbor green pixels to the green pixel currently being 
corrected is also a defective pixel. In the embodiment illustrated in Figures 3M through 
3T, the two nearest neighbor green pixels that are closest to the second defective pixel are 
weighted by 3/2 in the calculation of the numerical value for the identified defective 
pixel. 

[039] The embodiment of defective pixel correction illustrated in Figures 3A 
through 3T essentially utilizes numerical values for non-defective nearest-neighbor same- 
color pixels in a calculation to determine the numerical value of the identified defective 
pixel. Some embodiments of the invention may utilize further pixels in determining the 
numerical value of the defective pixel. For example, by utilizing numerical values for 
pixels fiirther removed fi'om the identified defective pixel, a numerical modeling routine 
may by utilized to determine a numerical value for the identified defective pixel. 
However, the more complicated is the computation, the slower is the process for 
determining the corrected numerical value for the defective pixel. 

[040] Li operation, defective pixel correct 105 reads the location of an 
identified defective pixel fi^om memory 104. The numerical values of nearest-neighbor or 
next-nearest-neighbor, non-defective pixels of the same color are then read fi"om memory 
104, as illustrated in Figures 3 A through 3T. A numerical value for the identified 
defective pixel is then calculated. The new numerical value can then be written into 
memory 104 to replace any value read for the identified defective pixel. Defective pixel 
correct 105 can operate on memory 104 concurrently with transfer of data from sensor 
interface 102 so long as all of the data required for a particular correction has been 
transferred to memory 104. 

[041] Figure 4 shows a block diagram of an ECIP block 106 according to the 
present invention. In accordance with the present invention, block 106 includes a lumina 
filter path 420 and a separate chroma filter path 422. Lumina filter path 420 and chroma 
filter path 422 separately operate on RGB data read from memory 104. 
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[042] ECIP block 106 includes a receive interface 401 for reading data from 
memory 104. In general, receive interface 401 reads NxN pixel blocks of data from 
memory 104. In some embodiments, data is written into memory by sensor interface 102 
in such a way that it can be easily read out as blocks of pixel data. For example, if the 
numerical values representing each pixel is 10 bits and the data is stored in 16 word 
blocks (128 bits), then blocks of, for example, 12 by 12 or 24 by 24 pixels are readily 
retrieved from memory 104. In some embodiments, then, N is 24 and 24x24 pixel blocks 
of data are input to ECIP block 106 for block processing. Receive interface 401 reads the 
pixel data in block format and unpacks the data to form the numerical values representing 
individual pixels in the block. 

[043] Black clamp adjust 402, which in Figure 4 receives the NxN pixel block 
data from receive interface 401, corrects each pixel value for a black value offset. In 
some embodiments, an optical value for each channel is calculated in black clamping 103 
(Figure 1). The values for each channel of pixel (i.e., the two green channels, the red 
channel and the blue channel), then, calculated by black clamping 103 can be subtracted 
in black clamp adjust 402. In some embodiments, black clamp adjust 402 can adjust the 
value for each pixel in the NxN data block by subtracting from it the offset value 
calculated for the channel of that pixel. In some embodiments, a set value stored in an 
offset register can be subtracted instead. In some embodiments, the choice between 
subtracting the calculated block offset value or the set black offset value can be 
programmed. In some embodiments, the numerical value representing each pixel may 
now have more bits and may be signed rather than unsigned. For example, if the 
numerical values for each pixel stored into memory 104 was 10 bit unsigned data, black 
clamp adjust 402 may output 1 1 bit signed data. 

[044] The new pixel data block generated in black clamp adjust 402 is input to 
AWB (all white balance) statistics 403. AWB statistics 403 calculates, for example, the 
average for each channel of pixel, the number of pixels in each channel with numerical 
values higher than a programmed limit, and the number of pixels in each channel with 
numerical values lower than a programmed limit. Statistical data can be stored in 
registers and held in latches between blocks of data. In some embodiments, the statistics 
compiled by AWB statics 403 can be utilized, after a number of blocks, for example 64 
blocks, of pixels have been processed. 
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[045] The new block data generated by black clamp adjust 402 is also input to 
white balance 404. White balance 404 adjusts the numerical values for pixels in each 
channel by a gain value in order to maintain a balance of data. In some embodiments, a 
constant average value for the numerical value in each channel is maintained. The 
numerical values for each of the pixels can be multiplied by a gain value designated for 
the channel of that pixel such that the average value for that channel calculated in AWB 
statistics 403 is at a particular value for that channel. 

[046] In some embodiments, where the numerical value for each of the pixels 
output from black clamp adjust 402 is 1 1-bit signed data, the numerical values for each of 
the channels is multipled by a 12-bit unsigned "gain coefficient" (i.e. Go = Go*Co, 
B =B*Ci, R =R*C2, and Gi ^Gi^Cs). In some embodiments, a "512" value for the 
coefficient is unity. The signed result can be rounded and descaled (i.e., bit-shifted to 
form a divide by 16) to 11 bit signed data. In some embodiments, an extra black offset 
subtract (C4) can be subtracted from the numerical value for each pixel. In some 
embodiments, the numerical value for each pixel can be low-range clamped at -128 and 
high-range clamped at a programmable value (C5). In some embodiments, the numerical 
values can be scaled by values obtained in AWB statistics 403. 

[047] The NxN pixel block of data output from white balance 404 is then input 
to both a lumina filter path 420 and a chroma filter path 422. Lumina filter path 420 can 
include edge enhancement, gamma correction, and spatial filters while determining the 
lumina Y numerical values. In some embodiments, lumina filter path 420 includes 
interpolation block 405, an RGB to Y conversion block 406, a gamma correction block 
408, an edge enhancement block 409, and a spatial filter 410. In some embodiments, 
chroma filter path 422 includes an interpolate block 41 1, a color correct block 412, a 
gamma correct block 413, a RGB to UV conversion block 414 and a core chroma block 
415. 

[048] As shown in Figure 5 A, interpolation block 405 of lumina filter path 420 
filters RGB numerical values for each of the pixels in the data block both vertically and 
horizontally to create three separate color planes, a red color plane, a green color plane 
and a blue color plane. In the three separate color planes, the filtering process provides a 
numerical value indicating the amount of that color, determined from surrounding pixels 
of the same color, for each pixel location in the plane. In positions where sensor array 
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101 does not have a pixel sensitive to that color, the numerical value for that color can be 
determined by nearby pixels of the appropriate color. For example, in the red color block 
numerical values indicating a red level will be determined for all locations even if that 
location in the pixel block originally received from memory 104 indicated a green or blue 
color and not a red color. 

[049] In some embodiments, numerical values indicating the amount of color at 
each pixel location in a color plane can be interpolated from nearest neighbor like-colored 
numerical values. In some embodiments, a multi-tap vertical and a multi-tap horizontal 
filter can be applied to interpolate for numerical values in all pixel locations. In general, 
filters are applied to the NxN block of numerical values of the pixel data block to 
determine numerical values of each pixel location indicating an amount of each color of 
the three color planes. 

[050] Figures 5A through 5D illustrate the fimction of one embodiment of 
interpolation block 405 of Figure 4. As shown in Figure 5 A, interpolation block 405 
includes a vertical filter 501 and a horizontal filter 502. In the embodiment shown in 
Figure 5A, vertical filter 501 is implemented before horizontal filter 502, however in 
some embodiments the reverse action may be performed. In yet other embodiments, the 
three-color planes created may be formed in one action based on surrounding values for 
nearest-neighbor same-colored pixels, similar to the calculations performed for defective 
pixels illustrated with Figures 3 A through 3T. 

[051] Figure 5B illustrates schematically execution of vertical filter 501 on an 
NxN block of RGB pixel data 504. In some embodiments of the invention, an M-tap 
filter is utilized. As illustrated in Figure 5B, M may be five (5). The output from vertical 
filter 501 includes three color specific color planes: a green plane 505, a red plane 506, 
and a blue plane 507. Each of color planes 505, 506 and 507 have a block size of (N- 
m)xN, with the number of rows reduced by m=(M-l). Where M is 5 and N is 24, for 
example, the 24x24 block input to vertical filter 501 emerges from vertical filter 501 as 
three 20x24 color planes, one for each of the three colors of pixels green, blue and red. 

[052] The M-tap filter usually is formed by M coefficients that weight the 
contribution of each of the pixels being utilized in the calculation. Only pixels of the 
color being filtered to form the color plane are utilized. Therefore, in forming green color 
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plane 505 only numerical values from green pixels in pixel block data 504 are utilized. 
Additionally, only data from red pixels in data block 504 are utilized to form red color 
plane 506 and only data from blue pixels in data block 504 are utilized to form blue color 
plane 507. 

[053] The coefficients of the M-tap filter can be chosen to suppress spatial 
frequencies above Nyquist (specifically 2F, where F is the Nyquist frequency), while 
providing as little attenuation as possible for frequencies below Nyquist, and slight 
overshoot on edges below V^ Nyquest. For example, in one embodiment a 5 -tap filter is 
utilized, with the coefficients being defined as (ti= -3,t2 = 8, t3= 22,t4= 8,t5 -3). In which 
case, the first five pixels in the first column of the example 24x24 block 504 maps to the 
value in the first-column, first-row position of filtered blocks 505, 506 and 507. The two 
red pixels of the first five pixels in the first column of the 24x24 block map into the first- 
row, first-column position of the red filtered block. Because there are no blue pixels 
included in the first five pixels of the first column of the 24x24 block, a zero value (or 
null) is mapped into the first-row, first-column position of the blue block. The second 
row, first column positions of the colored filtered blocks are determined by applying the 
5-tap filter to the second through sixth pixel values in the first column of the 24x24 block. 

[054] In general, the values of NxN block of data 504 are mapped through the 
M-tap filter to color planes 505, 506 and 507. M-taped filter can be expressed as (ti, t2, . . 
. tjvi). The mapping into color plane 505, then, can be expressed as 

M 

G =^t 

k=\ 

where Gj j indicates a numerical value in green color plane 505, i=l to (N-m), j= 1 to N, 
m=M-l and /J^. represents the numerical values of NxN block of data 504 with non- 
green pixel values being set to 0. Similarly, the mapping into color plane 506 can be 
expressed as 

M 
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where Rjj indicates a numerical value in the red color plane 506 and P/^j represents the 

numerical values of NxN block of data 504 with non-red pixel values being set to 0. 
Also, the mapping into color plane 507 can be expressed as 

A/ 

^ij ~ ^i+A:+m/2),y 

where Bij indicates a numerical value in the red color plane 506 and P^j represents the 

numerical values of NxN block of data 504 with non-red pixel values being set to 0. To 
optimize these calculations, coefficients need not be multiplied by non-existent data (i.e., 
for data from wrong-colored pixels). As illustrated in the example, although each 
position in the data block of green color plane 505 includes a numerical value indicating a 
level of green for that pixel, red color plane 506 and blue color plane 507 have altemating 
columns containing zero values because altemating columns of RGB data block 504 
contain only red and only blue pixels. 

[055] As indicated above, in some embodiments a 5 -tap filter is utilized with 
coefficients given by (ti, t2, ts, U, ts) = (-3, 8, 22, 8, -3). Other coefficient values can be 
utilized in the 5-tap filter. A high value for ts can sharpen responses, but may not provide 
accurate projections for pixels that are not of the color currently being filtered. The 
chosen values (-3, 8, 22, 8, -3) provide good line sharpening, a good fi^equency response, 
and the results can be easily divided by 16 in hardware to provide for easy rescaling. hi 
some embodiments, R, B, and vertically filtered G values are divided by 16 while 
horizontally filtered G values are divided by 32. 

[056] Figures 5C, 5D and 5E illustrate the operation of horizontal filter 502. 
Similarly to that described above with respect to vertical filter 501, each of color planes 
505, 506 and 507 are filtered to color planes 508, 509 and 510, respectively. Figure 5C 
illustrates the mapping of green color plane 505 into green color plane 508. Figure 5D 
illustrates the mapping of red color plane 506 into red color plane 509. Figure 5E 
illustrates the mapping of blue color plane 507 into blue color plane 510. 

[057] Although a different size filter may be utilized in horizontal filter 502, in 
some embodiments an M-tap filter is utilized for horizontal filtering. As a result, color 
planes 508, 509 and 510 have, as discussed above, m= M-1 fewer columns than color 
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planes 505, 506 and 507, respectively. In some embodiments, the same M-tap filter is 
utilized in both vertical filter 501 and horizontal filter 502. In the embodiment illustrated 
in Figures 5C through 5E, the same 5-tap filter that was utilized in vertical filter 501 is 
utilized in horizontal fiher 502. In embodiments with an M-tap filter described by (ti, t2, . 
. . tM), the numerical values of green color plane 508 can be expressed as 

M 

where Gi,i represents the numerical values of green color plane 505, i and j can have 
values fi-om 1 to (N-m), and m=M-l. Similarly, the numerical values of red color plane 
509 can be expressed as 

M 

where Ri,i represents the numerical values in red color plane 506, and the numerical 
values of blue color plane 510 can be expressed as 

M 
k^\ 

where Bi,i represents the numerical values in blue color plane 507. To optimize the 
calculations, the zero columns in red color plane 506 and blue color plane 507 may be 
omitted firom the calculation so that coefficients are not simply multiplied by zero. 

[058] Again, in some embodiments the same 5-tap filter can be applied in 
horizontal filter 502 as was applied in vertical filter 501 . In embodiments where a 24x24 
block of data is input to vertical filter 501, three 20x20 color planes are output at 
horizontal filter 502. The filtering has determined values for each position in the 20x20 
arrays in order to fill in for data in that color that was missed because pixels in that 
position in data block 504 were sensitive to other colors. Additionally, in some 
embodiments, the numerical values in color planes 508, 509 and 510 output fi-om 
horizontal filter 502 are 10-bit unsigned data with low range clamped at 0 and high range 
clamped at 1023. 
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[059] As shown in Figure 4, the red, green and blue color planes output from 
interpolate block 405, color planes 508, 509 and 510, are input to RGB to Y block 406. 
RGB to Y block 406 converts data from the three color planes 508, 509 and 510 to lumina 
data Y. RGB to Y 406 receives the three (N-m)x(N-m) color planes and calculates an (N- 
m)x(N-m) lumina block Y. The numerical values in the lumina block Y can be expressed 
as 

where the coefficients oto, ai, a2, and as can be user programmable. In some 
embodiments, the coefficients oco, ai, a2, and a3 can be 12 bit unsigned values, with the 
value 1024 being unity. The numerical values of Y (Yij) can be descaled with rounding 
and clamped at a low and high range, for example a low of 0 and high of 1023. The 
resulting numerical values of Y can be 10-bit unsigned data. 

[060] The numerical values of Y are input to statistics block 407. Statistics 
block 407 can calculate statistics utilized in an auto-focus feedback loop and an auto- 
exposure feedback loop, controlling other aspects of the image acquisition. The auto- 
focus (AF) statistical value can be the sum of the difference between adjacent luma pixel 
squared, or the absolute value. A small absolute value for the AF statistical value 
indicates on out-of-focus condition, the image depicted in the Y block is out of focus. 
The AF value, then, can be input to an auto focus circuit to control the optics in front of 
sensory array 101 (Figure 1). The AF value can be summed into a sum register and a 
latch circuit can be utilized to store the last frame (i.e., the last exposure of sensor array 
101) results. Auto focus is typically accomplished by averaging the AF value for a few 
frames, moving the focus motor, averaging the AF value for a few more frames, then 
looking at the difference in the AF values between measurements. In general, increasing 
AF values means focus is getting better and decreasing AF values means focus is 
becoming worse. The optimum technique for using the auto focus statistics are up to the 
customer's firmware. The statistics are generated by calculating the difference between 
horizontally adjacent Y values and squaring this value to obtain a value DifH=(Yy,x - 
Yy,x+i)^; calculating the difference between vertically adjacent Y values and squaring this 
value to obtain the value DifV=(Yy,x - Yy+i,x)^; and adding the values DifV and DifH to 
obtain DifSum. 
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[061] Additionally, auto-exposure (AE) values can be calculated in statistics 
block 407 by counting the number of pixels with Y values above or below high and low 
threshold values. The exposure time for sensor array 101 can be set utilizing the AE 
value. In some embodiments, the high and low threshold values can be programmable. 
AE values can be summed in count registers and latches can be utilized to hold the results 
between frames, 

[062] The Y data block from RGB to Y 406 is also input to gamma correct 408. 
Gamma correct 408 performs a distortion correction on the numerical values of the Y 
block based on the magnitude of the numerical value itself. The range of the numerical Y 
values are separated into a number of segments and a linear interpolation is performed on 
the numerical values of the Y block based on which segment the nmnerical value falls 
within, 

[063] The calculation can be performed by first characterizing the numerical 
value as belonging within a certain range. A subset of the bits of the numerical value is 
then utilized to look-up a particular segment in a user-provided look-up table in order to 
obtain an offset and a slope characterizing the linear segment. The table of slopes and 
offsets can be written into memory 104 (Figure 1). The numerical value is then replaced 
by the numerical interpolation. The output numerical values can be expressed as 

Y! J = offset{base) + slope(base) * {value of bits) , 

where the base is the identification of which segment the Yj j value falls within, the offset 
and slope are determined by the base, and the interpolation bits are some of the least 
significant bits of the numerical values for Y. 

[064] For example, in an embodiment where the numerical values Yjj of the Y 
block are 10 bit unsigned values ranging from 0 to 1023, a 64 segment piecewise linear 
interpolation can be defined as indicated in Table 1. As seen in Table I, in the range of 
values 0-31, 8 segments are defined and bits b4..b2 can be utilized to define which 
segment the value falls within. Bits b4..b2 can, therefore, be utilized to look-up the offset 
and slope values appropriate for that base in a look-up table written into memory 104. 
Bits bl..bO are the bits utilized for the interpolation. In some embodiments, the offset 
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value can include the numerical value at the beginning of the segment. In some 
embodiments, the least significant bits are replaced by the results of the calculation. 

[065] In some embodiments, the base value is 10 bits unsigned and the slope 
value is 10 bits unsigned. A value of 128, then, indicates a 1:1 slope. The output value, 
which then can replace the numerical values of the Y block, are 10 bit unsigned data. In 
some embodiments, the look-up table and Table I are writeable into memory 104 during 
operation. 



Table I 



Input 
Value 


#of 
Segments 


Bits used 
to select 
base 


Bits used 
to 

interpolate 


0-31 


8 


b4..b2 


bL.bO 


32-127 


12 


b6..b3 


b2..bO 


128-639 


32 


b8..b4 


b3..bO 


640-1023 


12 


b9..b5 


b4..b0 



[066] The luminance Y data block output from gamma correct 408 is input into 
edge enhance block 409. Edge enhance block 409 detects and enhances edges in order to 
sharpen the image designated by the Y block. The input data from gamma correct block 
408 to edge enhance block 409 is a (N-m)x(N-m) block of numerical values designating 
lumina, Y. In some embodiments, where the original pixel input block is 24x24 pixels 
and a 5-tap filter is utilized for both horizontal and vertical interpolation in interpolation 
block 405, the block of Y data input to edge enhance 409 is, then, a 20x20 block of data. 
In some embodiments, each numerical value is represented by a 10-bit unsigned value. 

[067] In edge enhance block 409, an edge can be detected utilizing a 3x3 filter 
on the block of Y data. In some embodiments, an edge detection kemel can be utilized as 
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the filter. In some embodiments, a low-pass filter can be applied. In some embodiments, 
the application of the particular filter can be programmed. 

[068] As illustrated in Figure 6 A, numerical values fi-om the Y block 601 are 
mapped into numerical values for the Y' block 602. Because a 3x3 filter is utilized 
during the mapping, the edge pixels (i.e., rows 1 and (N-m) and columns 1 and (N-m)) are 
lost in the mapping and Y' block 602 is a (N-m-2)x (N-m-2) block. If block 601 is a 
20x20 block, for example, then, the output fi-om edge enhance block 409 is an 18x18 
block. 

[069] In an embodiment that utilizes an edge detection kernel, the edge 
detection kemel can be expressed as 

■-1 -1 -T 
-18-1. 
-1 -1 -1 

The actual numerical values are chosen so that the sum of all contributions are 0. For 
each mappable numerical value in Y block 601, a value X is calculated fi:'om the above 
matrix. As such, for the mapping of Yij, for example, X can be expressed as 

jr = -i*y: -i*y -i*y -i*y +8*y 

_l*y _l*y -l*r -1*K 

The value X can then be compared with a programmable threshold value. If X is less 
than the threshold value, a value Yp can be set to 0. If X is greater than or equal to the 
threshold value, then Yp can be set to X*(a programmable gain coefficient). The value Yp 
can then be added to the numerical value Yjj that is currently being mapped to form Y'ij, 
i.e. 

where a represents the programmable gain coefficient. 

[070] In some embodiments, the value X can be represented by an 8 bit 
unsigned value, the value a can be represented by an 8-bit unsigned value, and the value 
of Yp can be represented by a 9-bit unsigned value. In some embodiments, Yp is clamped 
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between programmable low and high values. The output Y jj can then be rounded and 



represented by 8-bit unsigned data with a low clamped at 0 and a high clamped at 255. 

[071] In some embodiments, edge enhance 409 can implement a low pass filter 
followed by alpha-blending. Figure 6 illustrates the mapping of one numerical value Y 
from block 601 to a numerical value Y in block 602. The low pass filter can be 
expressed as the matrix 



The output of the low pass filter for numerical value Yjj of Y block 601, then, can be 
given by 



Again, the value of Yp can be added to Yjj to form Y'i_ij_i. The parameters pk,i of the low 
pass filter can be user defined and are designed to enhance edges. In some embodiments, 
the value of Y'ij is rounded and represented by an 8-bit unsigned data value with a low 
range clamped at 0 and a high range clamped at 255. 

[072] The data output fi^om edge enhance 409 is input to spatial filter 410. 
Spatial filter 410 further enhances the edge effects and is sensitive to various types of 
edges. A set of masks, each of which is sensitive to a type of edge, can be applied to sub- 
blocks of the Y data block to detect edges in the sub-block. Depending on the type of 
edge detected, a particular mapping of the sub-block can be applied. 

[073] Figure 6B shows a mapping of the (N-m-2)x(N-m-2) Y data block 603, 
which was received by spatial filter 410 from edge enhance 409, into filtered block 604. 
In some embodiments, the masks are 3x3 arrays. Therefore, 3x3 sub-blocks of Y data 



Po,-i Px,^x 
P-\,Q Po,0 P\,0 
P^IA Po,\ Pl,l 




A value Yp can then be computed as 



i;=/'o,o*«+(i-«)*^. 
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block 603 are mapped into filtered block 604. In some embodiments, the masks can be 
given by: 





' 0 


0 


0 




mask 1 = 


-1 


2 


-1 


9 




0 


0 


0 





which is sensitive to a horizontal line; 





0 


-1 0 


mask 2 = 


0 


2 0 




0 


-1 0 



which is sensitive to a vertical line; 





"-1 


0 


0 


mask 3 = 


0 


2 


0 




0 


0 


-1 



which is sensitive to a lefl-to-right downward diagonal line; 



mask 4 = 



0 0-1 
0 2 0 
-10 0 



which is sensitive to the right-to-left downward diagonal line. 



mask 5 = 



0 0 0 
0 2-1 
0-10 



which is sensitive to a left-top comer; 





" 0 


0 


0 


mask 6 = 


-1 


2 


0 




0 


-1 


0 



which is sensitive to a right-top comer; 



21 



mask 7 = 



0-10 
-12 0 
0 0 0 



which is sensitive to a right-bottom comer; and 





0 


-1 


0 


mask 8 = 


0 


2 


-1 




0 


0 


0 



which is sensitive to a left-bottom comer. In general, other masks can be utilized to 
detect various other shapes as well as these. 

[074] For an arbitrary 3x3 sub-block of Y-data block 603, each of the masks 
are applied to the sub-block. A mask, in general, can be depicted as 



mask = 



and can be applied to a 3x3 sub-block of Y data block 701 centered around Yjj, The 
resulting value can then be given by 

Value=Si*Yi.ij.i+S2*Yi.,j+S3*Yi.ij+,+S4*Yij.i+S5*Yij^^^ 

l+S8*Yi+ij+S9*Yi+ij+i. 

When values are compiled for all of the masks, the minimum value determines the filter 
that is applied to the 3x3 sub-block of Y data 603 centered around Yij. 



[075] Li some embodiments, the filters associated with each mask can be given 



by 



filter 1 = 



0 0 0 

1 2 1 
0 0 0 



which is applied if the value corresponding to mask 1 is the minimum value; 
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filter 2 = - 
4 



0 


1 


0 


0 


2 


0 


0 


1 


0 



which is appHed if the value corresponding to mask 2 is the minimum value; 



filter 3 = - 
4 



1 


0 


0 


0 


2 


0 


0 


0 


1 



which is applied if the value corresponding to meisk 3 is the minimum value; 



filter 4 = - 
4 



0 


0 


1 


0 


2 


0 


1 


0 


0 



which is applied if the value corresponding to mask 4 is the minimum value; 



filter 5 = - 
4 



0 


0 


0 


0 


2 


1 


0 


1 


0 



which is applied if the value corresponding to mask 5 is the minimum value; 



filter 6 = - 
4 



filter 7 = - 
4 



0 0 


0 




1 2 


0 


9 


0 1 


0 




mask 


6 is the minimiun value; 


0 1 


0" 




1 2 


0 


? 


0 0 


0 





which is applied if the value corresponding to mask 7 is the minimiun value; and 
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0 1 0 



filter 8 = - 
4 



0 2 1 



0 0 0 



which is applied if the value corresponding to mask 8 is the minimum value. 



[076] With the general equation for a filter being given by 



filter = 




f-\,0 f-\M 

/o,o /o.+i 

y+1,0 f+\M 



the mapped value for Yij can be given by 



In some embodiments, edge values for Yjj can be mapped directly, without a filter, to 
edge values of Y'ij. Therefore, filter block 604 is the same size as Y data block 603. In 
some embodiments, the edge values are not mapped so that the size of the filtered data 
block output firom spatial filter 410 is (N-m-4)x(N-m-4), or 16x16 for N=24 and m=4. 

[077] The output signal from spatial filter 410, which is now a (N-m-4) by (N- 
m-4) block of data, is rewritten back to memory 104 by transmission interface 416. In 
embodiments where a 5 -tap filter is utilized in interpolation block 405, and therefore 
m=4, and N=24, then the Y block of lumina data is an 16x16 block of lumina data Y. 

[078] The data output by white balance 404, the NxN pixel block, is input to 
interpolate block 41 1 of chroma filter path 422. Again three separate color planes, one 
for green, one for red, and one for blue, are determined based on the RGB data in the 
pixel block. Color values are determined for each pixel location in the block based on the 
numerical values for those colors in the RGB data. Although interpolation block 411 may 
be functionally of similar form to interpolation block 405 of lumina filter path 420, 
different considerations in data between lumina data and chroma data result in the use of 
different filters. 
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[079] In some embodiments, interpolation block 405 utilizes 5-tap filter in 
interpolation for both vertical and horizontal filtering. In some embodiments, 
interpolation block 411 utilizes a 7-tap filter. The coefficients of the 7-block filter can be 
chosen to improve moire rejection and suppress false color due to color aliasing. In 
general, more taps can be utilized in the interpolation 41 1 in chroma filter path 422 than 
in interpolation 405 in luma filter path 420 because the human eye is more sensitive to 
spatial distortion than it is to color distortion. 

[080] Figure 7 A shows a block diagram of interpolation 411 of chroma filter 
path 422. Interpolation 411 includes a vertical interpolation 701 and a horizontal 
interpolation 702. Although in the embodiment shown in Figure 7A vertical interpolation 
701 is executed before horizontal interpolation 702, in general the order can be reversed. 
Additionally, some other embodiments may utilize filters that interpolate the numerical 
value of a color at a pixel location based on numerical values of pixels of that color 
surrounding the location in both vertical and horizontal directions in one calculation. 

[081] Figure 7B illustrates vertical interpolation 701. In general, a K-tap filter 
can be utilized. Because the eye is less sensitive to color variations, the value of K can be 
greater than the value of M utilized for lumina interpolation in interpolation 405. In some 
embodiments, K=7. 

[082] Figure 7B illustrates the interpolation fi-om the NxN pixel block 703 
received by interpolation 411 into individual color planes 704, 705 and 706. To optimize 
the calculations, coefficients for filtering to one of color planes 704, 705 or 706 are not 
multiplied by numerical values in 703 that are not associated with that color. As 
illustrated, the size of color planes 704, 705 and 706 is reduced by k=K-l columns. As 
depicted in Figure 7B, green color plane 704 fills in all pixel locations with a green 
numerical value. However, because the blue and red pixels only occur every other 
column in pixel block 703, blue color plane 705 and red color plane 706 have alternating 
columns of zero or null values at completion of vertical interpolation 701. 

[083] In some embodiments, where N=24 and K=7, the pixel data block 703, 
which is a 24x24 block, is reduced in size to a 16x22 block in vertical interpolation 701. 
In chroma filter path 422, the outer edge of pixels is ignored because edge enhancement 
and spatial filtering need not be done, reducing pixel data block 703 to (N-2) x (N-2) (for 
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N=24, pixel data block 703 because a 22x22 block in vertical interpolation 701) initially. 
In some embodiments, the filter coefficients can be given by (2,5,6,6,6,5,2). For mapping 
pixel group 707, which contains 7 pixel values in a column, to the indicated position in 
green color plane 704, only the green values are applied. The value in 704, then, is 5 
times the first green value plus 6 times the second green value plus 5 times the last green 
value. The red value in red color plane 706 is given by 2 times the first red value plus 6 
times the second red value plus 6 times the third red value plus 2 times the fourth red 
value. Since there are no blue pixels in the second column of filter block 703, the 
corresponding value in blue color plane 705 is 0. Similarly, group 708 can be mapped to 
green color plane 704, red color plane 706 and blue color plane 705. 

[084] The filter values are chosen, as discussed above, to optimize moire 
rejection and suppress color aliasing. In addition, the values are picked for easy descaling 
(i.e., divide by 16). However, other filter values may be applied. 

[085] Once vertical interpolation 701 is complete, the three color planes 704, 
705 and 706 are then horizontally filtered in horizontal interpolation 702. In some 
embodiments, the same 7-tap filter that was applied vertically in vertical interpolation 701 
can now be applied horizontally on each of color planes 704, 705 and 706 in horizontal 
interpolation 702. 

[086] Figures 7C, 7D and 7E illustrate the horizontal interpolation. Green 
color plane 704 is mapped to green color plane 707; blue color plane 705 is mapped to 
blue color plane 708; and red color plane 706 is mapped to red color plane 709, As is 
illustrated, each of the color planes has a numerical value indicating that color in each 
pixel location. The size of color planes 707, 708 and 709 has then been reduced by k=K- 
1 both horizontally and vertically so that color planes 707, 708 and 709 are now (N- 
k)x(N-k)data blocks. 

[087] Where N=24 and K=7, color planes 707, 708 and 709 are 16x16 blocks. 
In some embodiments, the numerical values in color planes 707, 708 and 709 are 10-bit 
unsigned data low range clamped at 0 and high range clamped at 1023. 

[088] The output data fi-om interpolation 41 1, the three color planes 707, 708 
and 709, are input to color correction 412. In color correction 412, the numerical values 
are mixed to correct for color miss-interpretation in the original RGB data. As such, the 
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mappings of green color plane 707, blue color plane 708, and red color plane 709 in color 
correct block 412 can be expressed as 

R\j = * R.J + * G,j 4- A * B,j + A 
G]j = A * R,j + A * G,j -f A * B,j -f Ao > 

= fie * + A * + A * B, j + A, 

where the coefficients Pi through Pu are user programmable. In some embodiments, the 
coefficients are 12-bit signed values where "512" is unity. The output numerical values 
for each of the mapped red, green and blue color planes can be 10-bit unsigned data with 
low range clamped at 0 and high range clamped at 1023. The output data from color 
correct 412 is input to gamma correct 413. 

[089] In gamma correct 413, each of the red, green and blue color planes are 
processed independently utilizing 64 segment piecewise linear interpolation. Any non- 
linearity in response for the pixels can be corrected in this fashion. In some 
embodiments, RGB correct 413 may utilize the same gamma table as gamma correct 408, 
which is shown in Table L In some embodiments, a different segregation of segments 
may be utilized in gamma correct 413. In principle, however, the range determines the bit 
values to use for selecting a base value and a slope from a table that is stored in memory 
104, and the bits to be interpolated. For example, in the range 32-127 bits b6..b3 of the 
numerical value are utilized in a look-up table to obtain the base value and the slope for 
the appropriate segment and bits b2..b0 are interpolated based on that linear segment. 
Therefore, the output numerical value is given by 

Output=base+slope*(value of interpolation bits). 

[090] Different look-up tables can be utilized for mapping the red, green and 
blue color planes. Additionally, the look-up tables may be writable during operation. 
The gamma tables (an example of which is shown in Table I) can be separately loadable 
in memory 104 for luma filter path 420 and chroma filter path 422. 

[091] The corrected red, green and blue color planes output from gamma 
correct 413 are then input to RGB to UV conversion 414. In conversion block 414, the 
data for each color plane can be down-sampled by averaging blocks of data. In some 
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embodiments, 2x2 blocks of red, green, and blue color planes are utilized. In some 
embodiments, the down-sampling can be given by 

G;., = (G,, + G,,,,, + G,,,, + G,,,,,, ) / 4 
^.V =(^..> +^,...>.)/4 , 

=(^.v +^M.y +5w.,.,)/4 

where the primed values are the averaged values and the unprimed values are input to 
conversion block 414. The down-sampling average provides a 2:0 chroma, down 
sampled from 4:4. The output signal from conversion block 414, then, is an 8x8 block of 
data. 

[092] The U and V values are then calculated from the down-sampled averages 
computed above. Values for U and V can be given by 

= ""Kj +^4 ' 

The values of coefficients yo through y-j are programmable coefficients. In some 
embodiments, these coefficients may be 12-bit signed values, where 1023 is about 1, 
allowing for a maximum chroma gain of about 2. In some embodiments, the output U 
and V values are 12-bit signed data with low range clamped at -2048 and high range 
clamped at 2047. 

[093] The U and V data output from RGB to UV conversion 414 is input to 
core chroma 415. For each pixel in both the U data block and the V data block, the sum 
of the absolute values of the U numerical value and the V numerical value is calculated. 
If the sum is greater than a threshold value, the U and V values are unchanged. If the 
values are less than a threshold value, then U and V are each multiplied by the sum and a 
multiplier. Both the threshold value and the multiplier are programmable. In some 
embodiments, the threshold value can be an 1 1-bit unsigned value and the multiplier can 
be a 12-bit unsigned coefficient, where 4097 is about 1 . The output numerical values for 
U and V can be 8-bit signed data with a low clamped at -128 and a high clamped at 127. 

[094] The U and V data blocks output from chroma filter path 422 and the Y 
data block output from lumina filter block 420 are input to transmission interface 416, 
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where they are written into memory 104. As seen in Figure 1, these values may be 
filtered in filter 107 before encoded by encoding block 109. The final encoded data is 
transmitted through transmission interface 108. 

[095] An example of MPEG-4 Encoding 109 is provided in U.S. Application 
Serial No. 10/346,736, assigned to the same assignee as is the present invention, herein 
incorporated by reference in its entirety. 

[096] Other embodiments of the invention will be apparent to those skilled in 
the art from consideration of the specification and practice of the invention disclosed 
herein. It is intended that the specification and examples be considered as exemplary 
only, with a true scope and spirit of the invention being indicated by the following claims. 
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